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AUDIO TO VIDEO TIMING MEASUREMENT 
FOR MPEG TYPE TELEVISION SYSTEMS 



4 This application is a Continuation In Part to S/N 09/1 1 9,524 filed 07/2 1/98 which is a 

5 division of S/N 08/620,126 filed 03/21/96 which receives priority from provisional application 

6 60/008,309 filed December 7, 1995, which applications are incorporated herein in their entirety 

7 and for all purposes as if they had been set forth in detail. 

8 The examiner's attention is called to incorrectly published U.S. Patent 5,847,769 which is 

9 related to the present application by virtue of common application 08/620,126. The ' 769 patent 

10 was withdrawn from issue. Despite the fact of the patent being withdrawn from issue it was 

1 1 nevertheless published by the Patent Office. Applicant is of the belief that this mistakenly 

12 published patent does not constitute prior art, and is not available in respect to any double 

1 3 patenting matter or the like, but brings it to the attention of the examiner out of applicant's duty of 

14 candor. 
15 

1 6 BACKGROUND OF THE INVENTION 

1 7 The invention relates to measuring, maintaining and correcting synchronization between two 

1 8 signals which suffer varying relative delays during transmission and/or storage, and in particular to 

19 measuring the relative delay between multiple audio signals and an associated video signal of a 

20 television type program which is compressed via MPEG or other compression method for 

21 transmission and/or storage. 
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1 1 . FIELD OF THE INVENTION 

2 The present invention relates to the field of transmitting and storing multiple electronic 

3 signals where synchronization of the signals is of concern. When such transmitting and storing are 

4 of a nature which makes the corresponding receiving and recovering of said signals subject to 

5 timing errors resulting from differing amounts of processing delays the present invention is useful 

6 in measuring the relative timing errors or delays between signals with such delay measurement 

7 being used as a meter of quality of the transmitting and storing and for maintaining or correction 

8 of relative delays between such signals. 
9 

10 2. DESCRIPTION OF RELATED PRIOR ART 

11 It is known in the television signal transmission field to measure and correct audio to video 

12 timing errors by measuring the delay which a video signal experiences and using that measurement 

1 3 to delay a companion audio signal by a corresponding amount. 

14 U.S. Patent 4,313,135 by the present inventor shows to compare relatively undelayed and 

15 delayed versions of the same video signal to provide a delay signal responsive to the delay thereof 

16 and to couple that delay signal to a variable audio delay to cause the audio delay to delay the 

17 companion audio signal by a corresponding amount 

1 8 U.S. Patents 4,665,43 1 and 5,675,388 by the present inventor show transmitting an audio 

1 9 signal as part of a video signal so that both the audio and video signals experience the same 

20 transmission delays thus maintaining the relative synchronization therebetween. 

21 U.S. Reissue Patent RE 33,535 corresponding to 4,703,355 shows in the preferred 

22 embodiment to encode in the vertical interval of a video signal, a timing signal derived from an 
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1 audio signal and transmitting the combined video signal and the audio signal. At the receiving 

2 location the timing signal is recovered from the video signal and a new timing signal is generated 

3 from the received audio signal. The two timing signals are compared at the receiving location to 

4 determine the relative delay between the timing signal recovered from the video and the newly 

5 generated timing signal, thus determining the relative delay between the video and audio signals at 

6 the receive location. It is also suggested to put a timing signal in the audio signal. 

7 U.S. Patent 5,202,761 by the present inventor shows in the preferred embodiment to encode 

8 a pulse in the vertical interval of a video signal before the video signal is delayed. The encoded 

9 pulse is recovered from the vertical interval of the delayed video signal. Various methods 

10 responsive to the encoded pulse or the timing thereof for the undelayed video and the encoded 

1 1 pulse recovered from the vertical interval of the delayed video are shown which enable the 

12 determination of the delay, or the control of a corresponding audio delay. 

13 U.S. Patent 5,530,483 by the present inventor shows determining video delay by sampling 

14 an image of the undelayed video and sampling images, including the same image of the delayed 

15 version of the video and comparing the samples of the undelayed image to the samples of the 

1 6 delayed images until a match is found indicating that the undelayed image in delayed form is 

1 7 being compared. The time lapse between the sampling of the undelayed image, and the finding of 

1 8 the matching delayed image is used as a measure of video signal delay. 

19 U.S. Patent 5,572,261 by the present inventor shows a method of determining the relative 

20 delay between an audio and a video signal by inspecting the video for a speaker's mouth and 

21 determining various mouth patterns of movement which correspond to sounds which are present in 

22 the audio signal. The time relationship between a mouth pattern which creates a sound and the 
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1 occurrence of that sound in the audio is used as a measure of audio to video timing. 

2 U.S. Patent 5,751,368, a CIP of 5,530,483 shows the use of comparing samples of relatively 

3 delayed and undelayed versions of video signal images for determining the delay of multiple 

4 signals. 

5 Applicant incorporates all of the above prior art patents herein as fully as if they were set 

6 forth in their entirety for the purposes of enabling one of ordinary skill in the art to practice the 

7 present invention in so far as the present invention utilizes many elements which are taught 

8 therein. In particular, attention is called to RE 33,535 and the teachings of generating a timing 

9 signal in response to an audio signal, and the comparison of a recovered timing signal and a newly 

10 generated timing signal at the receiving site to determine the relative delay therebetween. 

1 1 The above cited inventions often prove to be less than complete solutions for modern 

12 television systems and others which transmit or store a plurality of signals for various reasons 

13 including for example those problems recited below. In particular, the current transmission of 

14 MPEG compressed television signals has proven to have particular difficulty in maintaining audio 

15 to video synchronization, and the prior art has particular problems in dealing with such. 

16 4,313,135 compares relatively undelayed and delayed versions of the same video signal to 

1 7 provide a delay signal. This method requires connection between the undelayed site and the 

1 8 delayed site and is unsuitable for environments where the two sites are some distance apart. For 

1 9 example where television programs are sent from the network in New York to the affiliate station 

20 in Los Angeles such system is impractical because it would require the undelayed video to be sent 

21 to the delayed video site in Los Angeles without appreciable delay, somewhat of an oxymoron 

22 when the problem is that the transmission itself creates the delay which is part of the problem. A 
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1 problem also occurs with large time delays such as occur with storage such as by recording since 

2 by definition the video is to be stored and the undelayed version is not available upon the 

3 subsequent playback or recall of the stored video. 

4 U.S. Patents 4,665,431 and 5,675,388 show transmitting an audio signal as part of a video 

5 signal so that both the audio and video signals experience the same transmission delays thus 

6 maintaining the relative synchronization therebetween. This method is expensive for multiple 

7 audio signals, and the digital version has proven difficult to implement when used in conjunction 

8 with video compression such as MPEG. 

9 U.S. Reissue Patent RE 33,535 corresponding to 4,703,355 shows in the preferred 

10 embodiment to encode a timing signal in the vertical interval of a video signal and transmitting 

1 1 the video signal with the timing signal. Unfortunately many systems strip out and fail to transmit 

12 the entire vertical interval of the video signal thus causing the timing signal to be lost. It is 

13 suggested to put a timing signal in the audio signal, which is continuous thus reducing the 

14 probability of losing the timing signal. Unfortunately it is difficult and expensive to put a timing 

15 signal in the audio signal in a manner which ensures that it will be carried with the audio signal, is 

16 easy to detect, and is inaudible to the most discerning listener. 

17 U.S. Patent 5,202,761 shows to encode a pulse in the vertical interval of a video signal 

1 8 before the video signal is delayed. This method also suffers when the vertical interval is lost. 

19 U.S. Patent 5,530,483 shows determining video delay by a method which includes sampling 

20 an image of the undelayed video. This method also requires the undelayed video, or at least the 

21 samples of the undelayed video, be available at the receiving location without significant delay. 

22 Like the ' 135 patent above this method is unsuitable for long distance transmission or time delays 
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1 resulting from storage. 

2 U.S. Patent 5,572,261 shows a method of determining the relative delay between an audio 

3 and a video signal by inspecting the video for particular sound generating events such as a 

4 particular movement of a speaker's mouth and determining various mouth patterns of movement 

5 which correspond to sounds which are present in the audio signal. The time relationship between a 

6 video event such as mouth pattern which creates a sound and the occurrence of that sound in the 

7 audio is used as a measure of audio to video timing. This method requires a significant amount of 

8 audio and video signal processing to operate. 

9 U.S. Patent 5,751,368, a CIP of 5,530,483 shows the use of comparing samples of relatively 

10 delayed and undelayed versions of video signal images for determining the delay of multiple 

1 1 signals. Like the '483 patent the '368 patent needs for the undelayed video or at least samples 

12 thereof to be present at the receiving location. 

1 3 Attempts have been made to add various timing related signals in television program 

14 streams in order to maintain audio to video synchronization. In particular in MPEG systems 

15 control signals such as time stamps are utilized. Unfortunately the inclusion of these signals does 

16 not guarantee proper audio to video synchronization at the receive side output of the system for a 

17 variety of reasons, including the fact that there are significant video delays which occur which can 

1 8 not be tracked by the time stamps. 
19 

20 BRIEF SUMMARY OF THE INVENTION 

21 It is an object of the invention to provide a method for measuring or maintaining the relative 

22 delay of a plurality of signals which are passed through subsequent processing. 
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1 It is another object of the invention to provide a method of generating a marker in response 

2 to a second signal which marker may be associated with a first signal in a fashion that said marker 

3 is carried with said first signal through processing of said first signal. 

4 It is still another object of the invention to provide a method of responding to a marker 

5 which has been associated with a first signal and a marker which is provided in response to a 

6 second signal whereby said markers may be utilized to determine the relative delay between said 

7 first and second signals. 

8 It is a further object of the invention to provide a marker in response to a signal wherein said 

9 marker indicates the occurrence of particular characteristics of said signal. 

10 It is a still further object of the invention to provide a system of measuring the relative delay 

1 1 between an audio and a video signal in a television system wherein the audio and video signals are 

12 subject to differing processing which creates unequal delays in said signals. 

13 It is yet still a further object of the invention to provide a method of marking a first signal 

14 which may be a video signal to allow relative delay measurement of said first signal and a second 

1 5 signal which may be an audio signal after they have been processed, including use of a marker 

16 generator responsive to the second signal to generate a marker upon the occurrence of one or more 

1 7 particular characteristics of the audio, associating the marker with the video signal in a fashion 

1 8 such that the marker will by carried with the video signal and not be adversely affected by the 

1 9 subsequent processing thereof. 

20 It is yet still another object of the invention to provide a relative delay measurement system 

21 for measuring the relative delay between a plurality of signals including a first signal which is a 

22 video signal and second signal which is an audio signal which signals experience unequal delays 
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1 due to processing thereof, the invention including use of a marker generator responsive to the 

2 audio signal to generate a marker upon the occurrence of one or more particular characteristics of 

3 the audio, associating the marker with the video signal in a fashion such that the marker will by 

4 carried with the video signal but not be adversely affected by the subsequent processing thereof, 

5 responding to the marker with the video signal after the processing to generate a first delayed 

6 marker; generating a second delayed marker in response to the processed audio signal, comparing 

7 the relative timing of the first and second delayed markers to determine the relative timing 

8 between the processed audio and processed video signal 

9 The preferred embodiment of the invention may be used with a television signal. At the 

10 transmitting location a marker is generated in response to the audio signal and is associated with 

1 1 the video signal such that the marker is carried with the video signal in a fashion such that it will 

12 not be lost or adversely affected by the expected processing of the video signal The audio signal 

13 and the marker associated video signal are stored, transmitted and/or processed and made available 

14 at a later time thus becoming delayed video and audio signals.. A first delayed marker is recovered 

1 5 from the delayed video signal and a corresponding second delayed marker is generated from the 

16 delayed audio signal, with the two delayed markers compared to determine the relative delay 

17 therebetween. This relative delay between these markers is responsive to and is a measure of the 

18 delay between the delayed video signal and delayed audio signal 

1 9 Somewhat simplistically stated, the preferred embodiment of the invention operates by 

20 generation of the marker at the transmit section, which may be thought of a marking the video at 

2 1 the time of the occurrence of a known event in the audio signal. The time marker is associated 

22 with the video signal such that it is carried in time with the video signal for all of the processing 
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1 which the video signal is to experience. After the video signal processing and any audio signal 

2 processing, the same event in the audio is again marked in time, and the previously marked time 

3 (relative to the video) is recovered or flagged in the received video. Since it is known that the 

4 audio event and the marking of the video occurred (substantially) simultaneously at the transmit 

5 location, the displacement between those events at the receive location is a measure of the audio to 

6 video timing error, or the relative delay therebetween. 

7 Generally, the present invention teaches measuring the relative delay between a plurality of 

8 signals which have suffered differing delays due to transmission, storage or other processing. The 

9 preferred embodiment of the invention includes the use of a marker which is generated in response 

1 0 to a second signal and combined with a first signal in a manner which ensures that the marker will 

1 1 not be lost in the expected processing of the first signal. Subsequently a first delayed marker is 

12 generated in response to the marker associated with or recovered from the first signal, and a 

13 second delayed marker is generated from the second signal. The first delayed marker and second 

14 delayed marker are compared to determine a measure of the relative timing or delay between said 

1 5 first signal and said second signal at said subsequent time. 
16 

1 7 BRIEF DESCRIPTION OF THE DRAWINGS 

1 8 Figure 1 shows a block diagram of the preferred embodiment of the invention as used with a 

19 television audio and video signal. 

20 Figure 2 shows a block diagram of the marker generator 3 and 13 of the preferred 

2 1 embodiment of the invention. 

22 
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1 DETAILED DESCRIPTION OF THE INVENTION 

2 In Figure 1 the preferred embodiment of the invention which is given by way of example, a 

3 video signal 1 and an audio signal 2 are present at what will be referred to as the transmit location. 

4 Either or both the video and audio signals may be in analog or digital, compressed or 

5 uncompressed form, the many variations and versions of which are well known in the art. Further, 

6 while the preferred embodiment is shown in respect to one video and one audio signal, it will be 

7 appreciated from the teachings herein that the invention may be utilized and practiced with 

8 multiple video and/or audio signals. In particular, by way of example the invention may be 

9 practiced with video and stereo (2 channel), surround (4+ channel) or 5.1 channel audio systems as 

1 0 are contemplated for the new U.S. digital and HDTV transmission standards. It is also noted that 

1 1 the components of the invention may be implemented by analog, digital or software means or 

12 combinations thereof. 

13 A marker generator 3 is responsive to the audio signal, and may be responsive to the video 

14 signal as indicated by the dashed line. In response to detecting the occurrence of one or more 

15 particular feature or characteristic of the audio signal generates a marker. One of ordinary skill in 

16 the art will recognize that element 44 of RE 33,535 may be utilized as element 3 herein. Other 

17 constructions and operations of 3 will also will be known to one of ordinary skill from the present 

1 8 teachings. The particular features, characteristics, occurrences or other event in the audio signal 

19 which will result in the marker, will be referred to hereinafter as occurrences and the marker in its 

20 various forms will sometimes be referred to simply as a marker, one of ordinary skill 

21 understanding from the context and the teachings herein the specificity of the form or forms being 

22 referred to. 
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1 The marker from 3 is associated with the video signal 1 in a marker associator 4. One of 

2 ordinary skill in the art will recognize that element 10 of the parent specification can be used for 

3 element 4 herein. Other constructions and operations of 4 will also will be known to one of 

4 ordinary skill from the present teachings. The marker is preferred to be associated with the video 

5 signal in a fashion that the marker will not be lost, corrupted or modified beyond use by 

6 subsequent processing of the video signal. In particular it is preferred to associate the marker with 

7 the video signal by including the marker within the active picture information of the video signal 

8 in one of the manners disclosed in detail in the parent specification. Consequently the marker may 

9 take on a form of active video, whatever form the video may be in. 

10 Alternatively, the marker may be associated with the video signal by being encoded in the 

1 1 active video in a relatively invisible fashion by utilizing one of the various watermark techniques 

12 which are well known in the art. Watermarking is well known as a method of encoding the 

13 ownership or source of images in the image itself in an invisible, yet recoverable fashion. In 

14 particular known watermarking techniques allow the watermark to be recovered after the image 

15 has suffered severe processing of many different types. Such watermarking allows reliable and 

16 secure recovery of the marker after significant subsequent processing of the active portion of the 

17 video signal. By way of example, the marker of the present invention may be added to the 

1 8 watermark, or replace a portion or the entirety of the watermark, or the watermarking technique 

19 simply adapted for use with the marker. It is believed that this use of watermarking techniques to 

20 associate marker signals with video signals for audio to video timing purposes is novel and 

21 previously unknown to those in the art. Other methods of associating the marker with the video 

22 signal will be known to those of ordinary skill in the art from the teachings herein. 
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1 The video signal with the marker is output from 4 and coupled to the video encoder 5. The 

2 video encoder 5 is used by way of example in the present description to represent that part of the 

3 subsequent video processing which may take place at the transmitting side of the system. For 

4 example, the video encoder may include MPEG preprocessing and compression circuits. 

5 Similarly, the audio 2 is coupled to an audio encoder 6 which is used by way of example in the 

6 present description to represent the audio processing which may take place at the transmitting side 

7 of the system. For example, the audio encoder may include an MPEG compression circuit. The 

8 compressed video and audio signals are combined by video and audio combiner 7 and the 

9 combined signals are coupled to the transmission channel 8. 

10 The audio and video signals from the transmission channel 8 are coupled to a video and 

1 1 audio separator 9 which separates the audio and video signal components of the transmitted 

12 sig^al(s). The audio and video signals are coupled to audio decoder 1 1 and video decoder 10 

13 respectively, where they are decoded back into decoded audio 17 and decoded video 16 

14 respectively. 

1 5 At the receiving side, marker separator 12 responds to the marker which was previously 

16 combined in the video signal by 4 to provide a first delayed marker to 14. The first delayed marker 

17 may be in the same form or different form as the marker which is associated with the video. It is 

1 8 preferred that the marker be recovered from the video and provided as the first delayed marker, 

19 however it is sufficient to merely detect the presence of the marker in the video and generate a first 

20 delayed marker in response thereto. One of ordinary skill in the art will recognize that element 40 

21 of the parent specification may be utilized for element 12 herein. Other constructions and 

22 operations of 12 will also will be known to one of ordinary skill from the present teachings. 
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1 Also at the receiving side, another marker generator 13, similar to 3, generates a second 

2 delayed marker in response to the same audio signal occurrences in the receive section audio from 

3 1 1 as the did the marker generator 3 on the transmit section in response to audio signal 2. Marker 

4 generator 13 may also be responsive to video in a fashion as previously described for 3 as shown 

5 by 19. The second delayed marker generated by 13 need not be in the same form as the marker 

6 generated by 3, but is preferred to be in the same form as the first delayed marker provided by 12. 

7 Th^first and second delayed markers from 12 and 13 are coupled to the relative timing 

8 comparison 14. The relative timing comparison is responsive to these delayed markers to 

9 determine the timing between corresponding pairs thereof to determine the relative timing between 

10 them, In other words the relative timing comparison 14 determines the delay 15 of the later of the 

1 1 two delayed markers relative to the earlier, indicating both the magnitude of the delay and which 

12 signal is more delayed. One of ordinary skill in the art will recognize from the teachings herein 

13 that relative timing comparison 14 may operate as described with respect to element 50 of the 

14 parent specification. Other constructions and operations of 14 will also will be known to one of 

1 5 ordinary skill from the present teachings. 

16 Since the first delayed marker from 12 experiences the delay of the video signal 1, and the 

17 second delayed marker from 13 experiences the delay of the audio signal 2 in their respective 

1 8 paths from the input of the transmit section to the output of the receive section, signal 1 5 is a 

19 measure of the relative delay of audio 17 and video 16 at the output of the receive section. 

20 The relative delay 15 may be utilized for all of the uses and reasons set forth in the parent 

21 specification. In particular note that the relative delay signal 15 is useful in itself as a measure of 

22 system quality. Relative delay signal 15 may be utilized to control a delay to delay the earlier of 16 
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1 or 17 to place the two signals into synchronization. Relative delay signal 15 may also be utilized to 

2 control a delay which is incorporated into 10 or 1 1 or both (or elsewhere in the system) to control 

3 the delay of the earlier of the audio or video from 9 to maintain the two signals 16 and 17 in 

4 synchronization. Relative delay signal 15 may also be utilized for other purposes, for example as 

5 feedback to control the operation of encoder 5 or 6 or decoder 10 or 1 1 to minimize or otherwise 

6 optimize delay or encoding and decoding of audio or video. 

7 Various different embodiments of the invention herein described will be apparent to one of 

8 ordinary skill in the art from the teachings herein. As an example, the marker generator 3 may be 

9 responsive to the video signal as shown by 1 8 in order to relate the marker to the video signal, for 

10 exrmple to properly locate the marker for combination with the video signal or to relate the 

1 1 particular feature(s) of the audio signal to timing of the video signal. In particular, it is desired 

12 that the marker represent whether or not the particular features occurred in the audio signal during 

13 the one or more frame or field immediately prior to the marker being combined with the video and 

14 going back to the time when the immediately previous marker was combined. 

1 5 The marker is preferred to be a binary signal which indicates that one or more of a number 

16 of particular occurrences of the audio which has taken place during the preceding field(s), or is 

1 7 currently taking place. For example, an 8 bit binary signal may be utilized with different numbers 

1 8 corresponding to different occurrences or features. In the preferred embodiment, it is preferred that 

19 the audio signal, which in the present example is assumed to have a bandwidth of 10 Hz to 20,000 

20 Hz be broken up into 8 different frequency bands by bandpass filtering. Each bit of the 8 bit 

21 number corresponds to the presence of audio frequencies within a particular band having energy 

22 within known levels and for known time durations. For example, if no such frequencies are 
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1 present, the binary number 0 (0000 0000) results. If the lowest frequencies occur, the binary 

2 number 1 (0000 0001) results. If the next highest frequency occurs the binary number 2 (0000 

3 0010) results. If both the lowest and next highest occur a 3 results. If all frequencies occur the 

4 binary number 255 (1 1 1 1 1 1 1 1) results. The binary number is the marker which is combined with 

5 the video. 

6 It is important to note that by associating the marker with the video signal in the fashion of 

7 including it in the active video portion of the signal that the marker will not be lost when all of the 

8 sync and blanking (or line, field and other ancillary signals if in digital form) are removed from 

9 the video signal such as is done as part of the MPEG encoding process. The association of the 

10 marker directly with the image carried by the video signal essentially guarantees that no matter 

1 1 what processing, stripping or modification of ancillary portions of the video signal occurs, in 

12 either analog or digital form, or conversion of scanning rates, or adjustment of usual video 

1 3 parameters such as black, brightness and chroma, that the marker will still be detectable at the 

14 receive location. 

15 The transmission channel 8 is utilized in the present example to represent any common or 

16 independent use or processing of the video signal 1 and audio signal 2 which may cause or result 

17 in unequal delays which lead to timing difficulties. Examples of such uses include transmission, 

1 8 storage and further processing, and in particular include storage and/or transmitting of MPEG 

19 encoded audio and video signals. 

20 Also it may be noted that marker generators 3 and 13 may respond to video in other forms, 

21 or from other parts of the system, or may respond to other signals, for example a genlock 

22 reference, in order to achieve proper operation and timing of the marker generator. 
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1 It may be noted that the use of the video encoder 5, audio encoder 6 and video and audio 

2 combiner 7 is given by way of example, as is usual for MPEG compression and transmission 

3 systems which are commonly used in today's television systems. The invention is not limited to 

4 the use of such elements however and one of ordinary skill in the art will know how to practice the 

5 generation of the marker and the associating of the marker with the video signal in other systems 

6 from the present teachings. The combined marker and video signal from 4 and the audio signal 2 

7 may very will be utilized in practicing the present invention without the added elements 5-7. 

8 1 1 will be understood that in the present example the elements 9, 1 0 and 1 1 are the receiving 

9 side elements complimentary to corresponding transmitting side elements 7, 5 and 6 respectively. 

10 As with 5, 6 and 7, elements 9, 10 and 1 1 are not required to practice the invention. In particular, 

1 1 video from 4 may be coupled, via a transmission channel directly to element 12 and become video 

12 signal 16. Similarly, audio signal 2 may be coupled via the same or different transmission channel 

13 directly to 13 and become audio signal 17. 

14 In the situation where the transmission channel includes storage of the audio and video 

1 5 signals, and storage and recovery is not performed simultaneously, it is noted that a single marker 

16 generator 3 may perform the function of 3 upon the storing of the signals and subsequently 

17 perform the function of 13 upon the recovery of the stored signals. Other sharing of circuitry 

1 8 between storing and recovery functions may also be had given the assumption that both are not 

19 performed simultaneously. 

20 Figure 2 shows the preferred form of the marker generator 3 and 1 3 of the preferred 

21 embodiment of the invention as used with television audio signals. Audio signal 20 which may 

22 correspond to 2 or the output of 1 1 in Figure 1 is coupled to a bank of 8 bandpass filters 2 la-h 
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1 which are configured to pass only audio within a range of frequencies as is well known in the art. 

2 The output of each bandpass filter is coupled to a comparator 22a-h respectively. The comparators 

3 include hysteresis or other threshold(s) and bipolar response characteristic so that if the positive or 

4 negative half cycle of bandpassed audio out of the bandpass filter exceeds a threshold amount set 

5 by the hysteresis, the output of the comparator is activated. Each comparator output is respectively 

6 coupled to a timing duration circuit 23a-h. Each timing duration circuit also receives a reset signal 

7 from the timing circuit 26. The timing circuit 26 provides signals to the parallel to serial converter 

8 24 in addition to the reset signal provided to the timing duration circuits 23. Once the timing 

9 duration circuit is reset, it inspects the output signal from its respective comparator 22. If the 

10 output signal from 22 is activated for an established time duration indicating the presence of audio 

1 1 frequencies within the corresponding bandpass filter range, the timing duration circuit sets its 

12 output active and holds it active until the next reset signal. The outputs of all of the timing 

13 duration circuits 23 are simultaneously latched into the parallel to serial circuit 24 upon command 

14 from the timing circuit 26 and shortly thereafter the reset signal to 23 is generated. Also shortly 

1 5 after latching, the bits latched into 24 are caused to be output in serial fashion as marker 25 . The 

16 net effect of the circuitry is to set a bit of the timing signal active corresponding to each of the 

17 bandpass audio frequencies which was present during the time period from one reset signal to the 

1 8 next, which corresponds to the time period from the generation of one marker to the next. The 

19 timing circuit 26 is responsive to the video signal to set the desired time period between markers, 

20 as well as to time the output of the marker 25 so that it is associated with the video signal at the 

2 1 correct time. This action will ensure that the marker is placed at the desired position in the video 

22 signal. 
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1 The bandpass filters are preferred to be selected to provide frequent outputs with the 

2 expected types of audio signals. For commercial television audio signals it has been found that 

3 bandpass filters with center frequencies of 25, 50, 150, 400, 1000, 2500, 6000, 15000 Hz and 

4 skirts of 6dB per octave work well. Other center frequencies and bandwidths may be chosen, and 

5 the number of filters changed, to facilitate expected audio signal frequency content. Ideally the 

6 frequencies would be chosen such that the lowest frequency filter has an output which is active or 

7 makes a change of state only once per period of the maximum expected delay differential of the 

8 audio and video signal. Alternatively, other audio characteristics may be relied on in the place of, 

9 or in addition to, the detection of energy at particular frequencies as described in respect to the 

10 preferred embodiment. Examples include, but are not limited to, impulse characteristics, amplitude 

1 1 characteristics, relationships between different frequency energies, relationships among and 

12 between different audio channels. 

1 3 Another example of alternate audio characteristics which may be utilized for the marker is 

14 the particular audio sonic characteristics which are relied on for the audio compression. Because 

15 these characteristics are already detected in the compression circuitry the present invention may 

16 share circuitry thus resulting in lowered cost. Other sharing of circuitry with other functions may 

17 be possible depending on the particular signals and environment with which the invention is used. 

1 8 While it has been described to utilize the marker generator with one audio signal in the 

19 preferred embodiment, it will be understood that multiple audio signals may be accommodated, 

20 with each having a corresponding marker which is associated with the video. Alternatively a 

21 plurality of audio signals may be used to generate a lesser number or even one marker by various 

22 techniques which include combining the plurality of audio signals before coupling to the marker 
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1 generator, or by combining various markers each responsive to one or a small number of audio 

2 signals with the various markers being combined into a smaller number or a single master marker. 

3 It may be noted that many audio ICs which are used for audio graphic equalizer functions 

4 contain bandpass filters which may be adapted to use in this invention. Of course it is possible to 

5 implement the various elements of the marker generator, as well as the rest of the invention, in 

6 analog or digital hardware, or software/hardware or combinations thereof. 

7 It will be noted that the present description of the preferred embodiment of the invention is 

8 given by way of example. In particular the diagrams of the preferred embodiment are presented as 

9 block diagrams and do not show in detail circuitry and cooperation which would be known to 

1 0 those of ordinary skill in the art from the teachings herein without undue experimentation. By way 

1 1 of example it is noted that where one signal line is shown in the block diagram that multiple 

12 signals may in actuality be coupled between one block and another, and although separate 

13 functional blocks are shown it will be known to make different combinations, arrangements and 

14 implementations in order to share elements therebetween and reduce costs. It is also noted that 

15 various terms used in the specification, including generator, combiner, encoder, separator, decoder 

16 and comparison, and their various tenses are intended to have broader meaning than that ordinarily 

17 ascribed thereto with respect to circuit elements, and are intended to cover not only the commonly 

1 8 understood element but the equivalent operation or function as implemented by other circuitry or 

19 software/hardware combinations. One of ordinary skill in the art will know to resort to various 

20 changes and modifications to the invention as described as well the combination of the invention 

21 with other features functions and/or inventive concepts in order to accommodate the use of the 

22 invention with particular forms of signals and otherwise to practice the invention in a fashion 
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1 which is optimized for particular application without departing from the spirit and scope of the 

2 invention as hereafter claimed. 
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What is claimed is: 

Claim 1 . In a system having a first signal which may be a video type signal and at least one 
second signal which may be an audio type signal the method of marking said first signal to 
facilitate later determination of the timing of said first signal relative to said second signal, said 
method including the steps of; 

marking said first signal with a marker in response to the occurrence of a particular event in 
said second signal said marking being such that said marker is carried with said first signal during 
the processing which said first signal is to experience. 

Claim 2. The method of determination of the timing of a first video type signal relative to its 
associated audio type signal after said audio type and video type signals have been processed, said 
method including the steps of: 

a) identifying the occurrence of a particular event in said audio type signal; 

b) in response to said occurrence of a), marking said video type signal with a marker such 
that said marking is carried with said video type signal for said processing which said video type 
signal is to experience; 

c) after said audio type signal processing, the occurrence of the same said particular event of 
a) in the processed version of said audio type signal is again identified; 

d) after said video type signal processing said marking of said video type signal of step b) is 
detected in the processed version of said video type signal; 

e) comparing the time at which said particular event is again identified in step c) to the time 
of said marking of said processed version of said video type signal being detected of step d) to 
determine the relative timing therebetween. 

21 



Claim 3. A relative delay measurement system for measuring the relative delay between a 
plurality of signals which experience unequal delays due to processing thereof said plurality of 
signals including a first signal and a second signal, said system including: 

a) a marker generator responsive to said second signal to generate a marker upon the 
occurrence of particular characteristics of said second signal; 

b) a marker associator responsive to said marker of a) and said first signal for marking said 
first signal with said marker in a fashion such that said marker will by carried with said first signal 
but not adversely affected by said subsequent processing; 

c) a marker separator responsive to said first signal after said subsequent processing to detect 
the presence of said marker therein and in response thereto generate a first delayed marker; 

d) a marker generator, which may be the same marker generator as in a), responsive to said 
second signal after said subsequent processing to generate a second delayed marker; 

e) a relative timing comparison responsive to said first delayed marker and said second 
delayed marker to determine the relative timing therebetween. 

Claim 4. The invention as claimed in any of the preceding claims wherein said first signal is 
a video signal and said second signal is an audio signal and said video signal is to be processed in 
a fashion which removes the horizontal and vertical blanking portions thereof. 

Claim 5. The invention as claimed in Claim 1, 2 or 3 wherein said first signal is a video 
signal and said second signal is an audio signal and said signals are to be transmitted in MPEG 
compressed form where only the active video portion of said video signal is transmitted. 

Claim 6. The invention as claimed in Claim 1, 2 or 3 wherein said first signal is a television 
video signal and said marking includes modifying the image portion of said video signal 
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Claim 7. The invention as claimed in Claim 1, 2 or 3 wherein said first signal is a television 
video signal and said marking includes modifying the image portion of said video signal by use of 
a watermarking technique. 

Claim 8. In a television system where a video signal and at least one audio signal are 
transmitted or stored in compressed form, the method of determination the timing of said video 
signal relative to said audio signal after said audio signal and said video and audio signals have 
been compressed and subsequently decompressed, said method including the steps of: 

a) identifying the occurrence of a particular event in said audio signal prior to compression; 

b) in response to said occurrence of a), marking the image area of said video signal with a 
marker; 

c) after said audio signal is decompressed, the occurrence of the same said particular event 
of a) is again identified in the decompressed version of said audio signal; 

d) after said video signal is decompressed detecting the presence of said marker in the 
decompressed version of said video signal; 

e) comparing the time at which said particular event is again identified in step c) to the time 
of detecting the presence of said marker of step d) to determine the relative timing therebetween. 

Claim 9. The method as claimed in claim 8 wherein said marking of step b) includes placing 
a binary number in the overscaned area of said image area. 

Claim 10. The method as claimed in claim 8 wherein said marking of step b) includes 
encoding a binary number in said image area as a watermark. 

Claim 1 1 . The method as claimed in claim 8 wherein said event of step a) is the presence of 
audio signal energy within a known frequency band. 
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Claim 12. The method as claimed in claim 8 wherein said event of step a) is the presence of 
a pattern of audio signal energy within a plurality of known frequency bands. 

Claim 13. The method as claimed in claim 1 1 or 12 wherein said audio signal energy must 
be greater than a known amount before being considered to be present. 

Claim 14. The method as claimed in claim 8 wherein said event of step a) comprises the 
occurrence of a known characteristic of said audio signal within a known period of time. 

Claim 15. The method as claimed in claim 8 wherein said event of step a) comprises the 
occurrence of a pattern of known characteristics of said audio signal within a known period of 
time. 
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Abstract 

An invention for measuring, maintaining and correcting synchronization between signals 
which suffer varying relative delays during transmission and/or storage is shown. The preferred 
embodiment of the invention finds particular use in measuring the relative delay between multiple 
audio signals and an associated video signal of a television type program which is compressed via 
MPEG or other compression for a transmission and/or storage system. The invention marks the 
video signal at a time when a particular event in the associated audio occurs. The mark is carried 
with the video throughout the video processing. After processing the same event in the audio is 
again identified, the mark in the video identified, the two being compared to determine the timing 
difference therebetween. 
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